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According to the present invention, a method for integrating 
processes with a multi- faceted human centered interface is 
provided. The interface is facilitated to implement a hands 
free, voice driven environment to control processes and 
applications. A natural language model is used to parse voice 
initiated commands and data, and to route those voice 
initiated inputs to the required applications or processes. The 
use of an intelligent context based parser allows the system 
to intelligently determine what processes are required to 
complete a task which is initiated using natural language. A 
single window environment provides an interface which is 
comfortable to the user by preventing the occurrence of 
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a plurality of facets which allow distinct viewing areas. Each 
facet has an independent process routing its outputs thereto. 
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itself to bring a new process into one of the viewing areas. 
All activated processes are executed simultaneously to pro- 
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METHOD FOR INTEGRATING COMPUTER Current computer technology allows application pro- 

PROCESSES WITH AN INTERFACE grams to execute their procedures within individual process 

CONTROLLED BY VOICE ACTUATED oriented graphical user interfaces (i.e. a "window"). Each 

GRAMMARS process is encapsulated in such a manner that all services 

5 required by the process are generally contained within the 

BACKGROUND OF THE INVENTION encapsulated process. Thus each object is an entity unto 

... , . c ... itself. Each process generally contains all of its own I/O 

The present invention relates to a method for controlling within its Qwn operating window . Whe n a process requires 

a series of processes with a human centered interface. More I/0> such as a keyboard inputj mouse input or the like, the 

precisely, the present invention relates to integrating a operating system passes the input data to the application or 

plurality of processes into a common user interface which is ™ Qb]QC{ h ^ conventionally known that a process window (a 

controlled by voice activated commands. The method fur- parem window ) spa wns a child window when the applica- 

ther includes a common framework which allows hands-free tion calls for speci fi c data entry (iy 0 ). This presents certain 

control of each process within the framework. A multitude pro bl e m S in that the child window does not release focus 

of processes can be easily integrated into the common from the child window until the child window is terminated, 

framework All processes which are controlled in the com- 35 when a keyboard and mouse are used as the primary 

mon framework can be executed in a multitasking environ- interface, the keyboard and mouse control will maintain 

ment - focus in the child window as long as the child window is 

Recent advances in computer technology has prompted an active. The viewing area becomes cluttered with child 

expansion in the use of personal computers for both business windows and it is difficult to read and parse all the infor- 

and home use. The widespread use of personal computers mation on the computer screen. 

has lead to a migration away from central based computing Current voice driven software technology is useful for 

on mainframes to distributed computing on personal com- utt i e more tDan a d i c tation system which types what is 

puters. Business applications often share common databases spo ken on a display screen. Although many programs have 

and system utilities across an interoffice network. With the ^ attempted to initiate command sequences, this involves an 

growth in the use of the internet, distributed computing extensive training session to teach the computer how to 

models have become increasingly important. By distributing handle specific words. Since those words are not maintained 

the resources necessary to accomplish a given task, the ^ a contex t based model that is intelligent, it is easy to 

amount of data required to be transferred across a network confuse such voice command systems. In addition, the 

can be reduced. 3q systems are limited in capability to the few applications that 

The desire to distribute processing and databases has support the voice interface, 
produced an industry of object based programming archi- One program, which was designed by the present 
tectures and languages. The proliferation of programming inventor, allows for voice activated commands to control a 
architectures/languages such as Java, Active X, C++, COM, user interface. This program (sold under the name VOICE 
OpenDoc and CORBA are a testament to this increased 35 PILOT™) contains a voice interface which allows for voice 
interest in distributed computing. Many prior art software initiated execution of programs as well as recording dicta- 
designs have been implemented on personal computers t i on . However, the overall architecture of this program 
based on these object oriented programming models. requires the use of child/parent windows as previously 

The Common Request Broker Architecture (CORBA) discussed. Every voice initiated application maintains its 

provides an object based programming architecture which 40 own operating window as a "child window" of the parent 

operates under a client/server topology. In a CORBA based process. The child window has to be satiated before releas- 

application program, every task is handled as an object ing control (active focus) and returning I/O access back to 

which is a self contained program. An Object Request the main program. 

Broker (ORB) serves as a mechanism for communicating The child/parent window configuration does not allow for 

client requests to target objects. Client requests appear as 45 complex command processing. A complex command 

local procedure calls. When a client invokes an operation, requires more than one process be performed in a specific 

the ORB finds the object, sends a request to the object and order based on a single spoken command phrase. For 

once the object completes the request returns any responses Example, the spoken command phrase "add Bob to address 

to the client. Each object operates independent of one book" is a multiple -step/multiple-process command. The 

another within the system. 50 appropriate commands required by the prior art are: "open 

In each object based programming model it is common address book", "new entry" and "name Bob". In the prior 

for each executing object to "pop -up" a "window" when any art, each operation is required to be completed one by one 

type of input or output (I/O) access is required by the user. in a sequential order. Although this methodology works to a 

When an object is executing a request, focus (an active minimum satisfaction level, it does not use natural language 

attention within its window) is granted to the object. Object 55 speech. The prior art is not capable of performing multiple 

oriented systems running on personal computers are gener- step operations with a single spoken command phrase, 

ally limited to a single active focus to a single object (within l n addition, the prior art does not provide that a single 

it's window) at any given time. spoken command phrase causes multiple processes to be 

Object based programming architectures like CORBA executed at the same time. For example, the spoken corn- 
provide very complex standards with which to work. A 60 mand phrase "Write a letter to Bob" requires multiple 
programmer must adhere to very stringent programming processes to be executed in order to effectuate the command, 
requirements in order to follow the CORBA standard. In The prior art would have to do the following: "open address 
order to allow multiple objects to be used together, CORBA book", "select Bob", "copy address", "open editor", "new 
uses a scripting language which queues objects in a letter" and "paste address". The address book and text 
sequence. A CORBA architecture does not permit parameter 65 editor/word processor are generally different applications, 
passing directly between objects and requires all parameters Since these programs require the data to be organized in a 
to pass through the common request broker. specific order, the voice commands must be performed in a 
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specific order to achieve the desired result. The prior art is 
not capable of performing operations simultaneously across 
multiple applications with a single spoken command phrase. 

Current computer technologies are not well suited for use 
with a voice driven interface. The use of parent and child 
windows creates a multitude of problems since natural 
language modeling is best handled with complex command 
processing. Since child windows receive active focus as a 
single window, they tend to sequentially process simple 
(single process) voice commands. 

The current invention seeks to overcome these limitations 
by providing a uniform speech aware interface that is 
optimized for a hands free, voice driven environment. This 
is especially useful for contact management, business pro- 
fessionals and anyone looking to eliminate the time wasting 
procedure of pushing and shoving windows around a video 
screen to find the useful data buried therein. By utilizing a 
voice interface, an innovative natural language processor 
and a unique graphical user interface which supports true 
multi-tasking, and I/O access which eliminates the use of 
"child" windows, the limitations of the prior art are over- 
come. 

OBJECTS AND SUMMARY OF THE 
INVENTION 

It is an object of the present invention to overcome the 
drawbacks in the prior art. 

It is an object of the present invention to provide a method 
of processing information using a hands free voice activated 
environment. 

It is a further object of the present invention to provide a 
universal access method to enable processes to connect to a 
common user interface. 

It is yet a further object of the present invention to provide 
a voice driven environment to control access to multiple 
processes. 

It is another object of the present invention to provide a 
method of parsing voice initiated utterances to establish 
context based decision making. 

It is another object of the present invention to provide a 
method of processing voice entered information by deter- 
mining if command and data elements are present within a 
phrase of the information. 

Briefly stated the present invention provides a method for 
integrating processes with a multi-faceted human centered 
interface is provided. The interface is facilitated to imple- 
ment a hands free, voice driven environment to control 
processes and applications. A natural language model is used 
to parse voice initiated commands and data, and to route 
those voice initiated inputs to the required applications or 
processes. The use of an intelligent context based parser 
allows the system to intelligently determine what processes 
are required to complete a task which is initiated using 
natural language. A single window environment provides an 
interface which is comfortable to the user by preventing the 
occurrence of distracting windows from appearing. The 
single window has a plurality of facets which allow distinct 
viewing areas. Each facet has an independent process rout- 
ing its outputs thereto. As other processes are activated, each 
facet can reshape itself to bring a new process into one of the 
viewing areas. All activated processes are executed simul- 
taneously to provide true multitasking. 

According to an aspect of the invention, there is provided 
for a method for controlling a plurality of processes by voice 
actuated grammars initiated by a user, each grammar having 
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at least one phoneme, the steps comprising: receiving an 
initial grammar from a process in response to the user 
initiating an utterance, setting a command mode of operation 
when the initial grammar from the step of receiving is 

5 determined to be a command activation statement, cycling 
through a first loop when in the command mode of 
operation, under control of the first loop: receiving a data 
stream from the process, the data stream containing at least 
one grammar, storing the data stream in a data storage 

10 location such that each the at least one grammar is in a 
separate location of the data storage location, searching the 
data storage location for a valid command statement, setting 
an error condition when the step of searching does not find 
the valid command statement, processing the valid com- 

15 mand statement when the step of searching finds the valid 
command statement, the valid command statement corre- 
sponding to at least one of the plurality of processes, and 
setting the mode of operation to a wait mode of operation 
when the step of processing the valid command statement is 

20 completed. 

According to another aspect of the invention, there is 
provided for a method for controlling a plurality of pro- 
cesses by voice actuated grammars initiated by a user, each 
grammar having at least one phoneme, the steps comprising: 

25 receiving an initial grammar from a process in response to 
the user initiating an utterance, the process including a 
speech-to-text processor, setting a command mode of opera- 
tion when the initial grammar from the step of receiving is 
determined to be a command activation statement, cycling 

30 through a first loop when in the command mode of 
operation, under control of the first loop: receiving a data 
stream from the process, the data stream containing at least 
one grammar, storing the data stream in a data storage 
location such that each the at least one grammar is in a 

35 separate location of the data storage location, searching the 
data storage location for a valid command statement, the 
step of searching includes comparing each the at least one 
grammar to a known vocabulary table, the vocabulary table 
containing a list of system commands and application com- 

40 mands which are registered in a process registration 
database, setting an error condition when the step of search- 
ing does not find the valid command statement, processing 
the valid command statement when the step of searching 
finds the valid command statement, the valid command 

45 statement corresponding to at least one of the plurality of 
processes, and setting the mode of operation to a wait mode 
of operation when the step of processing the valid command 
statement is completed. 

The above, and other objects, features and advantages of 

50 the present invention will become apparent from the fol- 
lowing description read in conjunction with the accompa- 
nying drawings, in which like reference numerals designate 
the same elements, 

55 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a general block diagram showing the flow of 
processes in a Multi-Faceted Human Centered Graphical 
Interface system according to the present invention. 
6Q FIG. 2 is a flow diagram showing a detail of the Initialize 
System step from FIG. 1. 

FIG. 3 is a flow diagram showing a detail of the Activate 
System step from FIG. 1. 

FIG. 4 is a flow diagram showing a detail of the Process 
65 User Input step from FIG. 3. 

FIG. 5 is a flow diagram showing a detail of the Parse 
Mode step from FIG. 4. 
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FIG. 6 is a flow diagram showing a detail of the Process a memory table (S205) and initializing the default applica- 

Stream step from FIG. 5. tions to the default viewing areas of the display (S206). 

FIG. 7 is a flow diagram showing a detail of the Process Once the initialization sequence ends (S120), program con- 
Command step from FIG. 4. trol is returned back to the Main process in FIG, 1. 

FIG. 8 is a flow diagram showing a detail of the Speech 5 It is important to note that there is only one "windowed" 

Activate Current Command Process step from FIG. 7 area of the display. The active viewing areas are all main- 

FIG. 9 is a flow diagram showing a detail of the Parse tained wit* 1 " 1 the confines of the single display area. Even 

Complex step from FIG. 8. when applications/processes are activated subsequently, the 

viewing area changes but no additional window areas ever 

DETAILED DESCRIPTION OF THE i° appear. Moreover, once an application is activated it remains 

PREFERRED EMBODIMENT active even when subsequent applications are activated. This 

FIGS. 1-9 contain a detailed flow chart for a multi-faceted P ermits a ^ e ™ ltitaskin S environment that is optimized for 

human centered interface (MFHCI) designed according to volce a PP hcatlons - 

the present invention. For convenience, every process step is The a *>ove initialization sequence (S200-S206) sets the 

designated with a process step identifier conatining a letter display screen area to multiple active viewing areas (called 

<S' followed by a three digit number (i.e. S300). Each facets). Each default application corresponds to one facet of 

process step ("Sxyz") uses a numbering convention where the display area. The output of the processes used by the 

the three digit code ("xyz") corresponds to the figure with application are directed to each applications corresponding 

which the process step relates. In every process step desig- facet. In a preferred embodiment of the invention, the 

nated as "SxOO", the "x" digit corresponds to the figure display screen contains three active facets corresponding to 

number in which the detail of the procedure is shown. In left, top and bottom portions of the display screen. Although 

each figure there is an end of process block which is it is understood that any number of facets (display portions 

designated as a "return" statement. Every "return" process of the view screen) can be utilized, it is preferred to use three 

step is designated as "SxyO", where V refers to which facets at a time. The human brain can easily handle three 

figure number to return to, and "y" refers to the current facets at a time without taxing the users. An excessive 

figure number. In all other process steps, the "x" refers to the number of facets creates a cluttered confusion to the user, 

current figure, and "yz" refers to a unique process step Likewise, the prior art methods are difficult for users to deal 

number which is none zero. The convention used to desig- with because parent/child screens appear one on top of 

nate process steps will become apparent from the discussion _ n another, creating an obscured, cluttered look to the screens, 

which follows In tne present invention, each screen portion corresponds to 

Referring to FIG. 1, there is shown a general block °ne of the active facets, which are all part of a single 

diagram for a multi-faceted human centered interface windowed environment. 

(MFHCI). The MFHCI functions as a universal human Referring again to FIG. 1, after the system is initialized in 
interface to allow enhanced access and usability to task 35 step S200, the system verifies that the speech mode is active 
management. By parsing a series of voice initiated (S102-104). This can be done, for example, by checking to 
utterances, a series of command sequences are identified and see if a microphone is present on the system and 
tasks are queued from the MFHCI. The voice initiated recognizable, and if no microphone can be found then the 
utterances can be any type of phoneme (the smallest unit of system disables the speech interface. When the speech 
speech that distinguishes one sound from another), word or 40 engine is disabled the keyboard and mouse function as the 
phrase in any language. In an alternative embodiment, a primary input device to the MFHCI (S105). The keyboard 
series of keyboard inputs can be used to initiate the com- and mouse are always active as secondary input devices 
mand sequence. In a preferred embodiment of the invention, when the speech interface is the primary interface, 
hands free voice initiated commands control the execution Once all the initialization parameters are set in place, the 
of processes, facilitate managing multiple tasks simulta- 45 system prompts the user (S106, FIG. 1) and goes into an 
neously and allows voice control of all applications. active mode (S300, FIG. 3). The prompt can be any 
The MFHCI is started by initiating a start command. The graphical/audio/visual prompt that is desired. For example, 
start command can be generated by speaking an utterance the TTS engine can be used by the system to announce a 
into a voice input device, striking a key on a keyboard, greeting. After the greeting/prompting has been made, the 
clicking a mouse input device on an icon, or any other so MFHCI system is active. The MFHCI system is running at 
known method of initiating execution of a sequence of all times. The voice/primary interface is actively monitored 
instructions. Once the start command is initiated, a main for possible commands, requests or data input. Even when 
process (S101, FIG. 1) is executed by the MFHCI. The main the system is in the middle of doing another process, the 
process: initializes a series of parameters (S200, FIG. 2), primary interface is being polled continuously for corn- 
verifies the speech engine is active (S 102-103) and sets the 55 naands. 

primary interface to be speech or defaults to keyboard and FIG. 3 shows a flow chart depicting the process flow 

mouse based upon the speech engine being active (S104 where the system has been activated by a user (S300). First, 

S105), prompts the user that the system is active (S106) and the user is prompted for input (S301). This prompting is not 

activates the system to accept all data I/O through the system a general greeting as in step S106 but instead is a request for 

(S300). 60 user input. This request can be in the form of any graphical, 

Referring to FIG. 2, a detail of the MFHCI initialization audio or visual cue which is necessary to alert the user that 

sequence (S200) is shown. This includes setting up a display the system expects an input. The system waits for the user 

screen area (S201), opening a series of previously desig- to generate an input (S302). The user generated input can be 

nated default applications (S202), loading a speech to text derived from any acceptable input process including, but not 

(STT) translation engine into the system (S203), loading a 65 limited to an SIT engine, a keyboard, a mouse or the like, 

text to speech (ITS) translation engine into the system Once an input is supplied, the MFHCI system processes the 

(S204), loading an active application dictionary (AAD) into input (S400, see FIG. 4 for details) and checks to see if the 
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input is a command to exit the system (S303). When the continually sent into the data store (S601). If the element in 

input is a request to exit the system (i.e. "Quit"), the main the data store is a valid command which is found in the 

process terminates (S130) and control returns back to the vocabulary table (S602-603) then the system process the 

main process in FIG. 1. Having received a request to exit, the command (S700). If the element was not in the vocabulary 

main process is subsequently terminated (S109). 5 table then it is deemed to not an invalid command statement, 

The general manner in which user input is processed is and the system sets the "success" flag to false (S604). Upon 

depicted in the flow chart found in FIG. 4. As shown in FIG. completion of the processing of the data stream, control is 

4, an input is processed (S400) by acquiring the data out of returned (S460) back to the Process User Input sequence 

a data stream (S401) and parsing that data elements to shown in FIG. 4. 

determine what operating mode the data stream requires for 10 Once the system has determined that a valid command is 

that data to be acted upon (S500, FIG, 5). The data stream contained in the data stream (S603, FIG. 6) the command 

can be found in a memory queue, a memory address, a must be processed. In order to process a command (S700) 

register, an operating system pipe, a shared memory area, on the system must perform a series of tasks as shown in FIG. 

a hard disk drive or any other means of passing data from an 7. The system compares the current command (CC) to a 

outside resource into a program. In instances where the 15 process registration table (not shown) in order to determine 

system has been integrated together with the input stream if the CC is part of a registered process (S701-702). All tasks 

processing program (i.e. a voice to text driver) the data (i.e. Application programs on a computer) that are controlled 

stream can be passed directly into the MFHCI as an internal by the MFHCI must be registered in the system in order to 

resource. In any case, the data elements which are received be useable by the MFHCI. After an application is registered 

from the input data stream corresponds to the data which is 20 in the system, the application is "known" to the MFHCI. A 

being supplied by the primary interface (i.e. the microphone registered application (RAP) is an application in which each 

driving the STT engine piping data out in text). The data is and every functional element has been described to the 

parsed to determine the mode of operation (S402). When it MFHCI. When an application is registered, the MFHCI 

is determined that the data stream requires a command mode amends the Known Vocabulary Table to contain a complete 

in order to be processed: the command mode is initiated 2 $ vocabulary for each functional aspect of the RAP. The 

(S403), the system processes the data stream (S600, details process registration also amends a known process table 

found in FIG. 6), the system checks to see if the data stream which cross-references applications/processes with known 

was successfully processed (S404), and if unsuccessful the vocabularies in the table. Thus, a dictionary of all the 

system prompts the user for more information (S405). When vocabulary required for all "known" applications is 

no command mode was initiated (S403) or upon completion 30 constructed, and a relational database or table allows the 

of processing the data stream, the system terminates the MFHCI to identify the vocabularies with the corresponding 

processing of user input (S340) and returns to the Wait mode processes, messages, events and methods. 

(S302, FIG. 3) or possibly exits the process (S303) as in FIG. For Example, one application which lends itself to voice 

3- commands is a telephone dialer program. During registra- 

FIG. 5 shows a flow chart depicting the parsing of an input 35 tion of the telephone dialer program, a series of words or 

data stream in order to determine if the system needs to be phrases are registered with the MFHCL The MFHCI amends 

in the command mode (S500). The stream is first divided the vocabulary table to contain the required vocabulary 

into a series of grammars which place the phrases in a relevant to the telephone dialer program. Examples of the 

context based data structure (S501) which is stored in a data required vocabulary could be keywords and phrases such as: 

store memory. The data store is searched for a Command 4 q "dial", "call", "connect", "hang-up", "disconnect", "hold", 

Activation Statement (CAS). When a CAS is found in the "transfer", "forward", "redial" and the like. Once the 

data store (S503), the system is set into command mode vocabulary is contained in the table, the MFHCI will con- 

(S504), otherwise the system defaults into a non-command struct a relational table to identify those vocabularies with 

mode. the telephone dialer program and all its functional features. 

A CAS is a unique word/phrase/keystroke or the like 45 As shown in FIG. 7, after the CC has been determined to 

which alerts the system that a user has an instruction waiting be a known vocabulary corresponding to a registered 

for service. In a preferred embodiment the user speaks a process, then the system determines if the CC is associated 

unique word into the computer microphone which alerts the with a process which is already activated (S703-S704). 

computer that commands will follow hereafter. For When a CC is associated with an active process, the active 

Example, the computer can be named, when the name of the 50 process is set to one of the facets of the display and the facet 

computer is spoken in a context which is unique (i.e. a pause, is speech activated (S800). If a CC is associated with a 

followed by "HAL", followed by a pause) then in that process that is not active, the user is prompted to select a 

context, the system knows that it has been alerted of the facet (viewing area within the single "window"), the process 

CAS. Similarly, a unique phrase can indicate a CAS (i.e. the is loaded into memory, the process is associated with the 

spoken phrase "Activate Voice"). Once the CAS is uttered, 55 selected facet and the facet is speech activated (S705, S706, 

the system goes into an active mode. It is important to note S800). Once an application or process is loaded into 

that the CAS can be contained in a dialog where notes are memory, it is active even though it may or may not be 

being recorded that are unrelated to the activation of the attached to one of the facets. All applications that are 

system (i.e. "Hal needs a new modem"). In instances where activated remain active until terminated. Since the user 

the context is unrelated to the activation command, the 60 dictates to the system, and the system locates the processes 

computer will not mistake this for a CAS. By reviewing the for the user, it is not necessary for the active process to be 

locus of words around a possible CAS, context is determined in the viewing area for it to be active. As the context of the 

and an appropriate action is taken. user input changes from one process to another, the system 

Once the system enters the command mode (S403 in FIG. dynamically "morphs" the facets of the display to accom- 

4), the system processes the incoming data stream (S600) as 65 modate what is needed to accomplish the task. By morphing 

shown in FIG. 6. The vocabulary table, which was loaded the facet (dynamically changing the contents of the viewing 

during initialization, is compared to the data stream which is area without effecting the surrounding viewing areas) the 
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users concentration does not get broken up with a series of 
keystrokes, moving windows to free up space and the like. 

The present invention grants a focus (an active attention) 
to the current facet which corresponds to the CCP when 
speech is activated with respect to this process. This is 
known as granting SPOCKUS (Speach Operational Control 
Kernal User Service) to the CCP. The CCP is enabled to 
interact with the Speech Aware Interface (SWI). All speech 
data that is streamed into the system is analyzed within the 
context of CCP (the SPOCKUS which has been directed to 
the CCP). Although the CCP is granted SPOCKUS, all 
facets and all processes are still known to the system to be 
active. Since all processes are still active, the SWI can 
"pipe" data to any one of the active processes if it is 
determined that the incoming data stream should be routed 
into a particular process. 

As shown in FIG. 8, the speech activated process receives 
streamed data (S801), parses the stream in Parse Complex 
process (S900) and determines if the streamed data is a CCP 
Command (S802), A CCP Command is a command that is 
known to the CCP that has been granted SPOCKUS. If the 
system doesn't identify a CCP command (S803-S804) then 
the system places the incoming stream in a buffer to be either 
discarded or placed in a data field of the current CCP, based 
on the context of the data. When a CCP command is 
identified by the system, the system determines (S805) if the 
command requires the process to terminate the process (i.e. 
"Done" or "Close File") or to execute a command in the 
CCP (S806). The system continuously processes the incom- 
ing data until it is instructed to terminate. 

The analysis and parsing of the incoming data stream is 
related to the current command process. For example, a 
contact management program might be activated by uttering 
"Open the calendar and show me today's appointments". 
This utterance is parsed (divided up into application/ 
computer related pieces) as a command to "open calendar" 
which initiates a CCP (the calendar process), and a com- 
mand to "show appointments" which is a CCP command. 
"Today" is parsed as a data field for the CCP Command. As 
described above, the incoming data stream is analyzed to 
determine which process the stream should be routed to, 
opening the required application program if it is not cur- 
rently active, and initiating a command which is recogniz- 
able by the contact manager which is now the current 
command process. 

As discussed above, since all processes are still active, the 
SWI can "pipe" data to any one of the active processes. 
Thus, the system branches into other CPs without losing the 
current CP (CCP). Since the SWI is always active and 
monitoring the input data stream, the system can receive a 
command which is outside of the CCP. Upon receiving this 
command, the system activates another CCP, granting 
SPOCKUS to the new CCP, and executes the other process. 
When the subsequent process is terminated, the system 
returns SPOCKUS back to the previous CCP. The system 
can perform this recursive type of behavior over a plurality 
of CCPs. 

Referring to FIGS, 8 and 9, each utterance received from 
the stream is parsed to determine the context and to refine 
the data elements using the Parse Complex (S900) proce- 
dure. If the CCP is a Dictation Active Command (DAQ then 
we must determine the context of the stream (i.e. a note 
taking program is active which receives dictation as its data 
elements). As shown in steps S901-S904, the incoming 
stream is parsed by checking the context of the adjacent 
words to the possible command word. A natural linguistic 
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model is used to determine if the possible command word is 
within the scope of the adjacent words. The linguistic model 
is used to break apart verbs, nouns, adjectives etc. If the 
possible command word is within the scope of those words, 

5 then the command word is within the context of the dictation 
and the words are passed through as data elements (S905). 
If the possible command word is outside the scope of those 
words, then the command word is in fact a new command. 
We check the new command to determine if it relevant 

10 within the CCP. If the new command is authenticated then 
the new command is passed back to the Speech Activate CC 
Process (S890). Similarly, when we are not in the Dictation 
state (a non DAC Command is the CCP), the system 
authenticates the command as a new command, and passes 

is it back to the Speech Activation process if it is valid (S902, 
S906^S908, S890). 

Having described preferred embodiments of the invention 
with reference to the accompanying drawings, it is to be 
understood that the invention is not limited to those precise 

20 embodiments, and that various changes and modifications 
may be effected therein by one skilled in the art without 
departing from the scope or spirit of the invention as defined 
in the appended claims. 
What is claimed is: 

25 1. A method for controlling a plurality of processes by 
voice actuated grammars initiated by a user, each grammar 
having at least one phoneme, the steps comprising: 
receiving an initial grammar from a process in response to 
said user initiating an utterance; 

30 

setting a command mode of operation when said initial 
grammar from said step of receiving is determined to 
be a command activation statement; 

cycling through a first loop when in said command mode 
35 of operation; 

under control of said first loop, 

receiving a data stream from said process, said data 
stream containing at least one grammar, storing said 
data stream in a data storage location such that each 
40 said at least one grammar is in a separate location of 
said data storage location, 
searching said data storage location for a valid command 
statement, 

45 setting an error condition when said step of searching 
does not find said valid command statement, 
processing said valid command statement when said step 
of searching finds said valid command statement, said 
valid command statement corresponding to at least one 
50 of said plurality of processes, and 

setting said mode of operation to a wait mode of operation 
when said step of processing said valid command 
statement is completed. 

2. A method for controlling a plurality of processes as in 
55 claim 1, wherein said step of receiving a grammar from a 

process is a step of receiving a grammar from a speech-to- 
text processor. 

3. A method for controlling a plurality of processes as in 
claim 2, wherein said step of searching said data storage 

60 location for a valid command statement is a step of com- 
paring each said at least one grammar to a known vocabulary 
table. 

4. A method for controlling a plurality of processes as in 
claim 2, wherein said step of searching said data storage 

65 location for a valid command statement is a step of com- 
paring each said at least one grammar to a known vocabulary 
table. 
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5. A method for controlling a plurality of processes as in 
claim 2, wherein said step of searching said data storage 
location for a valid command statement is a step of com- 
paring each said at least one grammar to a known vocabulary 
table, said vocabulary table containing a list of system 5 
commands and application commands which are registered 

in a process registration database. 

6. A method for controlling a plurality of processes as in 
claim 1, wherein said step of searching said data storage 
location for a valid command statement is a step of com- 10 
paring each said at least one grammar to a known vocabulary 
table. 

7. A method for controlling a plurality of processes as in 
claim 6, wherein said step of searching said data storage 
location for a valid command statement is a step of com- 15 
paring each said at least one grammar to a known vocabulary 
table, said vocabulary table containing a list of system 
commands and application commands. 

8. A method for controlling a plurality of processes as in 
claim 6, wherein said step of searching said data storage 20 
location for a valid command statement is a step of com- 
paring each said at least one grammar to a known vocabulary 
table, said vocabulary table containing a list of system 
commands and application commands which are registered 

in a process registration database. 25 

9. A method for controlling a plurality of processes as in 
claim 1, wherein said step of searching said data storage 
location for a valid command statement is a step of com- 
paring each said at least one grammar to a known vocabulary 
table, said vocabulary table containing a list of system 30 
commands and application commands. 

10. A method for controlling a plurality of processes as in 
claim 9, wherein said step of searching said data storage 
location for a valid command statement is a step of com- 
paring each said at least one grammar to a known vocabulary 35 
table, said vocabulary table containing a list of system 
commands and application commands. 

11. A method for controlling a plurality of processes as in 
claim 1, wherein said step of searching said data storage 
location for a valid command statement is a step of com- 
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paring each said at least one grammar to a known vocabulary 
table, said vocabulary table containing a list of system 
commands and application commands which are registered 
in a process registration database. 

12. A method for controlling a plurality of processes by 
voice actuated grammars initiated by a user, each grammar 
having at least one phoneme, the steps comprising: 

receiving an initial grammar from a process in response to 
said user initiating an utterance, said process including 
a speech-to-text processor; 
setting a command mode of operation when said initial 
grammar from said step of receiving is determined to 
be a command activation statement; 
cycling through a first loop when in said command mode 

of operation; 
under control of said first loop, 

receiving a data stream from said process, said data 

stream containing at least one grammar, 
storing said data stream in a data storage location such 
that each said at least one grammar is in a separate 
location of said data storage location, 
searching said data storage location for a valid com- 
mand statement, said step of searching includes 
comparing each said at least one grammar to a 
known vocabulary table, said vocabulary table con- 
taining a list of system commands and application 
commands which are registered in a process regis- 
tration database; 
setting an error condition when said step of searching 

does not find said valid command statement, 
processing said valid command statement when said 
step of searching finds said valid command 
statement, said valid command statement corre- 
sponding to at least one of said plurality of processes, 
and 

setting said mode of operation to a wait mode of 
operation when said step of processing said valid 
command statement is completed. 
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